Record the phase times a timed-out model reached - #345
Merged
Conversation
omc's alarm now asks the running command to unwind before it kills the process, so a command that runs out of time still reports how far it got. Both watchdogs have to outwait that grace, or they kill omc in the middle of producing the answer: `sendExpressionTimeout`'s own deadline and test.py's outer per-model budget. `shared.alarmGrace` mirrors `SystemImpl__alarm`'s formula. The simulation phase is also recorded on every way out, not just the ones that reach a handler. `sendExpressionTimeout` ends the process itself when omc dies, so the caller's `except TimeoutError` never runs; `runningPhase` covers that, and now takes the results dict so a later runner's own simulation time survives it too. `simElapsed` falls back to the wall clock when omc reports no time of its own, which is what an aborted run does. In the run of 2026-08-30 this was 90 models and 23936 s of wall clock recorded against no phase at all, 34 of them reporting `finalphase=0` with every clock at zero after 300-800 s. The `master` lane, whose simulations are separate processes this script kills itself, had none. The ladder that turns omc's nested stopwatches into per-phase times also had two guards swapped: the backend's own time was computed under `templates != -1` and SimCode's under `simcode != -1`, each testing the clock of the wrong phase. A translation that died in SimCode was recorded as having died in the backend, and the phase before the failure got no time. It is now one ladder over the clock that reads -1, which is both what says how far the translation got and what says which differences can be taken. Assisted-by: Claude Opus 5
sjoelund
enabled auto-merge (squash)
August 31, 2026 08:58
sjoelund
disabled auto-merge
August 31, 2026 09:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
omc's alarm now asks the running command to unwind before it kills the process, so a command that runs out of time still reports how far it got. Both watchdogs have to outwait that grace, or they kill omc in the middle of producing the answer:
sendExpressionTimeout's own deadline and test.py's outer per-model budget.shared.alarmGracemirrorsSystemImpl__alarm's formula.The simulation phase is also recorded on every way out, not just the ones that reach a handler.
sendExpressionTimeoutends the process itself when omc dies, so the caller'sexcept TimeoutErrornever runs;runningPhasecovers that, and now takes the results dict so a later runner's own simulation time survives it too.simElapsedfalls back to the wall clock when omc reports no time of its own, which is what an aborted run does.In the run of 2026-08-30 this was 90 models and 23936 s of wall clock recorded against no phase at all, 34 of them reporting
finalphase=0with every clock at zero after 300-800 s. Themasterlane, whose simulations are separate processes this script kills itself, had none.The ladder that turns omc's nested stopwatches into per-phase times also had two guards swapped: the backend's own time was computed under
templates != -1and SimCode's undersimcode != -1, each testing the clock of the wrong phase. A translation that died in SimCode was recorded as having died in the backend, and the phase before the failure got no time. It is now one ladder over the clock that reads -1, which is both what says how far the translation got and what says which differences can be taken.Assisted-by: Claude Opus 5